Image processor and image processing method

ABSTRACT

An image processor includes: an image sensor which outputs a short exposure image and a long exposure image; a sensor controller which controls first and second exposure sensitivities; a motion blending ratio calculator which calculates a motion blending ratio based on a motion amount of a subject; a motion-adapted image synthesizer which generates a motion-adapted image based on the motion blending ratio; and an HDR image synthesizer which synthesizes the motion-adapted image and the short exposure image, to generate an HDR image. The sensor controller controls the first exposure sensitivity: so that the first exposure time changes from increasing to constant and the first sensor gain changes from constant to increasing, when the brightness of the subject decreases from a first subject brightness; and so that the first exposure time changes to increasing and the first sensor gain changes to constant, when the brightness decreases from a second subject brightness.

FIELD

The present disclosure relates to an image processor and an imageprocessing method.

BACKGROUND

Image processors disclosed in Patent Literatures 1 and 2 have been knownas techniques of generating a high dynamic range (HDR) image having awide dynamic range by synthesizing a plurality of images of subjectscaptured in different exposure times.

In the image processor of Patent Literature 1, a reference image andother images are compared with each other so as to detect dynamic areasincluded in the images. Furthermore, in the image processor, a dynamicarea included in the reference image is replaced with a correspondingarea of another image so as to generate a replacement image. An HDRimage is generated by synthesizing the replacement image and thereference image. In this way, the HDR image is generated inconsideration of a moving object.

In the image processor in Patent Literature 2, a synthesis ratio of aplurality of images captured with different exposure times and to besynthesized is determined based on saturation degrees of areas in whichmotions of a subject have been detected, when the plurality of images ofthe subject are synthesized. In this way, an HDR image with a reducednoise influence, etc. is generated.

CITATION LIST Patent Literature

-   Patent Literature 1: Japanese Unexamined Patent Application    Publication No. 2011-188277-   Patent Literature 2: Japanese Unexamined Patent Application    Publication No. 2016-139876

SUMMARY

However, the image processors according to the related art can beimproved.

In view of this, the present disclosure provides an image processor andan image processing method which enable further improvement.

An image processor according to an aspect of the present disclosure isan image processor which generates a high dynamic range (HDR) image of asubject. The image processor includes: an image sensor which outputs, inone-frame time, (i) a first image of the subject captured with a firstexposure time and a first sensor gain and (ii) a second image of thesubject captured with a second exposure time and a second sensor gain,the second exposure time being longer than the first exposure time; asensor controller which, when a brightness of the subject changes, (i)controls a first exposure sensitivity so that the first image has afirst image brightness and (ii) controls a second exposure sensitivityso that the second image has a second image brightness, the firstexposure sensitivity being a product of the first exposure time and thefirst sensor gain, the second exposure sensitivity being a product ofthe second exposure time and the second sensor gain; a level adjusterwhich adjusts a luminance level of the first image to substantiallymatch a luminance level of the second image, to generate a correctedimage from the first image; a motion amount detector which detects amotion amount of the subject based on a difference in pixel valuebetween pixels co-located in the corrected image and the second image; amotion blending ratio calculator which calculates a motion blendingratio based on the motion amount, the motion blending ratio being aratio between the corrected image and the second image when thecorrected image and the second image are blended; a motion-adapted imagesynthesizer which synthesizes the corrected image and the second imagebased on the motion blending ratio, to generate a motion-adapted image;and an HDR image synthesizer which synthesizes the motion-adapted imageand the first image, to generate the HDR image. The sensor controller:controls the first exposure sensitivity so that the first exposure timechanges from increasing to constant and the first sensor gain changesfrom constant to increasing, when the brightness of the subjectdecreases from a first subject brightness; and controls the firstexposure sensitivity so that the first exposure time changes fromconstant to increasing and the first sensor gain changes from increasingto constant, when the brightness of the subject decreases from a secondsubject brightness lower than the first subject brightness.

It is to be noted that these general and specific aspects may beimplemented using a system, a method, an integrated circuit, a computerprogram, or a recording medium such as computer-readable compactdisc-read only memory (CD-ROM), or any combination of systems, methods,integrated circuits, computer programs, or recording media.

The image processor, etc. according to the aspect of the presentdisclosure enables further improvement.

BRIEF DESCRIPTION OF DRAWINGS

These and other objects, advantages and features of the disclosure willbecome apparent from the following description thereof taken inconjunction with the accompanying drawings that illustrate a specificembodiment of the present disclosure.

FIG. 1 is a block diagram illustrating a configuration of an imageprocessor according to Embodiment 1.

FIG. 2 is a diagram for illustrating processing performed by a sensorcontroller of the image processor according to Embodiment 1.

FIG. 3 is a flow chart indicating a flow of an operation performed bythe image processor according to Embodiment 1.

FIG. 4A is a schematic view of an example of a long exposure imageaccording to Embodiment 1.

FIG. 4B is a schematic view of an example of a short exposure imageaccording to Embodiment 1.

FIG. 4C is a schematic view of an example of a corrected short exposureimage according to Embodiment 1.

FIG. 4D is a schematic view of an example of a motion amountdistribution of a subject detected by a motion amount detector accordingto Embodiment 1.

FIG. 4E is a schematic view of an example of an HDR image generated bythe image processor according to Embodiment 1.

FIG. 5 is a diagram for showing processing performed by a sensorcontroller of an image processor according to a comparative example.

FIG. 6 is a diagram showing a long exposure image, a corrected shortexposure image, and an HDR image which are generated by the imageprocessor according to the comparative example when the brightness of asubject is illuminance L4.

FIG. 7 is a diagram showing a long exposure image, a corrected shortexposure image, and an HDR image which are generated by the imageprocessor according to the comparative example when the brightness ofthe subject is illuminance L5.

FIG. 8 is a diagram showing a long exposure image, a corrected shortexposure image, and an HDR image which are generated by an imageprocessor according to an example when the brightness of a subject isilluminance L4.

FIG. 9 is a diagram showing a long exposure image, a corrected shortexposure image, and an HDR image which are generated by the imageprocessor according to the example when the brightness of the subject isilluminance L5.

FIG. 10 is a block diagram illustrating a configuration of an imageprocessor according to Embodiment 2.

FIG. 11 is a diagram for illustrating processing performed by a sensorcontroller of the image processor according to Embodiment 2.

DESCRIPTION OF EMBODIMENTS Underlying Knowledge Forming Basis of thePresent Disclosure

The Inventors have found that the image processors described in theBACKGROUND section entail problems indicated below.

The image processors described above entail a problem that motion blurmay occur in an HDR image when a moving object is imaged under lowilluminance, for example, at evening or night.

An image processor according to an aspect of the present disclosure isan image processor which generates a high dynamic range (HDR) image of asubject. The image processor includes: an image sensor which outputs, inone-frame time, (i) a first image of the subject captured with a firstexposure time and a first sensor gain and (ii) a second image of thesubject captured with a second exposure time and a second sensor gain,the second exposure time being longer than the first exposure time; asensor controller which, when a brightness of the subject changes, (i)controls a first exposure sensitivity so that the first image has afirst image brightness and (ii) controls a second exposure sensitivityso that the second image has a second image brightness, the firstexposure sensitivity being a product of the first exposure time and thefirst sensor gain, the second exposure sensitivity being a product ofthe second exposure time and the second sensor gain; a level adjusterwhich adjusts a luminance level of the first image to substantiallymatch a luminance level of the second image, to generate a correctedimage from the first image; a motion amount detector which detects amotion amount of the subject based on a difference in pixel valuebetween pixels co-located in the corrected image and the second image; amotion blending ratio calculator which calculates a motion blendingratio based on the motion amount, the motion blending ratio being aratio between the corrected image and the second image when thecorrected image and the second image are blended; a motion-adapted imagesynthesizer which synthesizes the corrected image and the second imagebased on the motion blending ratio, to generate a motion-adapted image;and an HDR image synthesizer which synthesizes the motion-adapted imageand the first image, to generate the HDR image. The sensor controller:controls the first exposure sensitivity so that the first exposure timechanges from increasing to constant and the first sensor gain changesfrom constant to increasing, when the brightness of the subjectdecreases from a first subject brightness; and controls the firstexposure sensitivity so that the first exposure time changes fromconstant to increasing and the first sensor gain changes from increasingto constant, when the brightness of the subject decreases from a secondsubject brightness lower than the first subject brightness.

According to this aspect, the sensor controller controls the firstexposure sensitivity so that the first exposure time changes fromincreasing to constant and the first sensor gain changes from constantto increasing when the brightness of the subject decreases from thefirst subject brightness. In this way, the first exposure time is keptto be short, for example, even when the moving subject is imaged underlow illumination. Thus, it is possible to reduce motion blur in thefirst image. As a result, the motion blur in the motion-adapted imageobtained by synthesizing the corrected image and the second image isalso reduced. Thus, it is possible to reduce motion blur in the HDRimage. In addition, when the brightness of the subject decreases fromthe second subject brightness, sensor controller controls the firstexposure sensitivity so that the first exposure time changes fromconstant to increasing and the first sensor gain changes from increasingto constant. In this way, the first sensor gain is kept to be small, forexample, even when the moving subject is imaged under extremely lowilluminance. Thus, it is possible to reduce noise in the first image. Asa result, the noise in the motion-adapted image obtained by synthesizingthe corrected image and the second image is also reduced. Thus, it ispossible to reduce noise in the HDR image.

For example, the sensor controller may be configured to control thesecond exposure sensitivity so that the second exposure time changesfrom increasing to constant and the second sensor gain changes fromconstant to increasing, when the brightness of the subject decreasesfrom a third subject brightness lower than the first subject brightness

According to this aspect, the sensor controller performs control so thatthe first exposure time changes from increasing to constant at a timingearlier than the timing at which the second exposure time changes fromincreasing to constant when the brightness of the subject becomesdecreases. In this way, the first exposure time is kept to be short at acomparatively early timing. Thus, it is possible to effectively reducemotion blur in the first image.

For example, the sensor controller may be configured to control thefirst exposure sensitivity so that the first exposure time changes fromincreasing to constant and the first sensor gain changes from constantto increasing, when the brightness of the subject decreases from afourth subject brightness lower than the second subject brightness.

As described above, the first exposure time changes from constant toincreasing and the first sensor gain changes from increasing to constantwhen the brightness of the subject decreases from the second subjectbrightness. For this reason, for example, when the moving subject isimaged under extremely low illuminance, noise in the HDR image isreduced, but motion blur may occur in the HDR image. According to thisaspect, the sensor controller performs control so that the firstexposure time changes from increasing to constant when the brightness ofthe subject decreases from the fourth subject brightness. In this way,increase in the first exposure time is reduced. Thus, it is possible toreduce motion blur in the HDR image.

An image processing method according to an aspect of the presentdisclosure is an image processing method for generating a high dynamicrange (HDR) image of a subject. The image processing method includes:(a) outputting, in one-frame time, (i) a first image of the subjectcaptured with a first exposure time and a first sensor gain and (ii) asecond image of the subject captured with a second exposure time and asecond sensor gain, the second exposure time being longer than the firstexposure time; (b) when a brightness of the subject changes, (i)controlling a first exposure sensitivity so that the first image has afirst image brightness and (ii) controlling a second exposuresensitivity so that the second image has a second image brightness, thefirst exposure sensitivity being a product of the first exposure timeand the first sensor gain, the second exposure sensitivity being aproduct of the second exposure time and the second sensor gain; (c)adjusting a luminance level of the first image to substantially match aluminance level of the second image, to generate a corrected image fromthe first image; (d) detecting a motion amount of the subject based on adifference in pixel value between pixels co-located in the correctedimage and the second image; (e) calculating a motion blending ratiobased on the motion amount, the motion blending ratio being a ratiobetween the corrected image and the second image when the correctedimage and the second image are blended; (f) synthesizing the correctedimage and the second image based on the motion blending ratio, togenerate a motion-adapted image; and (g) synthesizing the motion-adaptedimage and the first image, to generate the HDR image. In (b): the firstexposure sensitivity is controlled so that the first exposure timechanges from increasing to constant and the first sensor gain changesfrom constant to increasing, when the brightness of the subjectdecreases from a first subject brightness; and the first exposuresensitivity is controlled so that the first exposure time changes fromconstant to increasing and the first sensor gain changes from increasingto constant, when the brightness of the subject decreases from a secondsubject brightness lower than the first subject brightness.

According to this aspect, the first exposure sensitivity is controlledso that the first exposure time changes from increasing to constant andthe first sensor gain changes from constant to increasing when thebrightness of the subject decreases from the first subject brightness.In this way, the first exposure time is kept to be short, for example,even when the moving subject is imaged under low illumination. Thus, itis possible to reduce motion blur in the first image. As a result, themotion blur in the motion-adapted image obtained by synthesizing thecorrected image and the second image is also reduced. Thus, it ispossible to reduce motion blur in the HDR image. In addition, the firstexposure sensitivity is controlled so that the first exposure timechanges from constant to increasing and the first sensor gain changesfrom increasing to constant when the brightness of the subject decreasesfrom the second subject brightness. In this way, the first sensor gainis kept to be small, for example, even when the moving subject is imagedunder extremely low illuminance. Thus, it is possible to reduce noise inthe first image. As a result, the noise in the motion-adapted imageobtained by synthesizing the corrected image and the second image isalso reduced. Thus, it is possible to reduce noise in the HDR image.

It is to be noted that these general and specific aspects may beimplemented using a system, a method, an integrated circuit, a computerprogram, or a recording medium such as computer-readable CD-ROM, or anycombination of systems, methods, integrated circuits, computer programs,or recording media.

Hereinafter, embodiments are described in detail with reference to thedrawings.

It is to be noted that each of the embodiments described below describesa general or specific example. The numerical values, shapes, materials,constituent elements, the arrangement and connection of the constituentelements, steps, the order of the steps, etc. indicated in the followingembodiments are mere examples, and do not limit the present disclosure.In addition, among the constituent elements in the following exemplaryembodiments, constituent elements not recited in any one of theindependent claims that define the most generic concept are described asoptional constituent elements.

Embodiment 1 1-1. Configuration of Image Processor

First, a configuration of image processor 2 according to Embodiment 1 isdescribed with reference to FIGS. 1 and 2. FIG. 1 is a block diagramillustrating the configuration of image processor 2 according toEmbodiment 1. FIG. 2 is a diagram for illustrating processing of sensorcontroller 6 of image processor 2 according to Embodiment 1.

As illustrated in FIG. 1, image processor 2 includes image sensor 4,sensor controller 6, level adjuster 8, motion amount detector 10, motionblending ratio calculator 12, motion-adapted image synthesizer 14,luminance blending ratio calculator 16, and HDR image synthesizer 18.

It is to be noted that image processor 2 is applied, for example, as amonitoring camera for imaging number plates of vehicles. Alternatively,image processor 2 may be applied to, for example, digital still cameras,digital video cameras, on-board object detection systems, on-boardelectronic mirrors, and on-board driving recorders.

Image sensor 4 is a so-called line-by-line image sensor which outputs,in one frame time, a short exposure image (an example of a first image)and a long exposure image (an example of a second image) of a subjectcaptured with varying exposure times and sensor gains. Specifically,image sensor 4 outputs the short exposure image of the subject capturedwith a first exposure time and a first sensor gain and the long exposureimage of the subject captured with a second exposure time and a secondsensor gain. Here, the second exposure time is longer than the firstexposure time.

Each exposure time means a time from start to end of charge accumulationperformed by a photoelectric conversion element of image sensor 4according to incident light. The sensor gain means the gain factor(amplification factor) of an amplifier for amplifying an image signal inimage sensor 4. The exposure times or sensor gains are changed so as tochange an exposure sensitivity which is the product of the exposure timeand the sensor gain.

Line-by-line image sensor 4 starts output of a short exposure imageimmediately after completion of output of a long exposure image in eachof first to last lines of a frame. In short, long exposure images andshort exposure images are alternately output in one frame time.

As illustrated in FIG. 1, a long exposure image output from image sensor4 is input to sensor controller 6, motion amount detector 10,motion-adapted image synthesizer 14, and luminance blending ratiocalculator 16. Moreover, a short exposure image output from image sensor4 is input to sensor controller 6, level adjuster 8, luminance blendingratio calculator 16, and HDR image synthesizer 18.

Sensor controller 6 controls the first exposure time, the first sensorgain, the second exposure time, and the second sensor gain of imagesensor 4. Specifically, as illustrated in (a) and (b) in FIG. 2, sensorcontroller 6 performs control so that the first exposure sensitivity andthe second exposure sensitivity increase as the brightness of thesubjects decreases. The first exposure time is the product of the firstexposure time and the first sensor gain, and the second exposuresensitivity is the product of the second exposure time and the secondsensor gain. In other words, when the brightness of the subject changes,sensor 6 controls the first exposure sensitivity so that the shortexposure image has a first image brightness and controls the secondexposure sensitivity so that the long exposure image has a second imagebrightness.

As illustrated in (a) and (b) in FIG. 2, when the brightness of thesubject decreases from illuminance L1 (one example of a first subjectbrightness), sensor controller 6 controls the first exposure sensitivityso that the first exposure time changes from increasing to constant (T1)and the first sensor gain changes from constant (G1) to increasing.Furthermore, when the brightness of the subject decreases fromilluminance L2 (one example of a second subject brightness) lower thanilluminance L1, sensor controller 6 controls the first exposuresensitivity so that the first exposure time changes from constant (T1)to increasing and the first sensor gain changes from increasing toconstant (G2 (>G1)). In this case, a preset limit value for the firstexposure time is T2 (>T1).

In addition, as illustrated in (a) and (b) in FIG. 2, when thebrightness of the subject decreases from illuminance L3 (one example ofa third subject brightness) lower than illuminance L1 and higher thanilluminance L2, sensor controller 6 controls the second exposuresensitivity so that the second exposure time changes from increasing toconstant (T3 (>T1)) and the second sensor gain changes from constant(G1) to increasing. In this case, a preset limit value for the secondexposure time is G3 (>G1).

It is to be noted that, in the examples illustrated in (a) and (b) inFIG. 2, illuminance L1 is approximately 200 Lux, illuminance L2 isapproximately 25 Lux, and illuminance L3 is approximately 100 Lux.

Level adjuster 8 adjusts the luminance level of the short exposure imageoutput from image sensor 4 to substantially match the luminance level ofthe long exposure image, to generate a corrected short exposure image(an example of a corrected image) from the short exposure image.Specifically, level adjuster 8 raises the gain of a signal indicatingthe short exposure image according to the ratio of the second exposuresensitivity and the first exposure sensitivity, so that the luminancelevel of the short exposure image rises to or close to that of the longexposure image. Level adjuster 8 outputs the generated corrected shortexposure image to motion amount detector 10 and motion-adapted imagesynthesizer 14.

Motion amount detector 10 detects a motion amount, which indicates theamount of a motion of a subject, based on the corrected short exposureimage and the long exposure image. Motion amount detector 10 includessubtractor 20, absolute value calculator 22, and block differenceaccumulator 24.

Subtractor 20 calculates a difference between the long exposure imageoutput from image sensor 4 and the corrected short exposure image outputfrom level adjuster 8. Specifically, subtractor 20 outputs a differencethat is a result of subtracting the pixel value of a pixel in thecorrected short exposure image from the pixel value of a pixel in thelong exposure image. The pixels are located at mutually correspondingpositions. Subtractor 20 performs such subtraction on each of all thecorresponding pixels of the long exposure image and the corrected shortexposure image. Subtractor 20 outputs the calculated difference toabsolute value calculator 22.

Absolute value calculator 22 calculates the absolute value of thedifference from subtractor 20. The absolute value of the differenceindicates an amount of a pixel value difference between the pixelsco-located in the long exposure image and the corrected short exposureimage. Absolute value calculator 22 outputs the calculated absolutevalue of the difference to block difference accumulator 24.

Block difference accumulator 24 accumulates the absolute values ofdifferences from absolute value calculator 22 for each image block. Theimage block is a unit when the overall area of the long exposure image(or corrected short exposure image) is split into, for example, n×npixels (n≥2). The larger the motion of the subject in a second exposuretime is, the larger the absolute value of a difference from absolutevalue calculator 22 is. The absolute value is increased by motion blurof a subject image in the long exposure image. In other words, thelarger the motion of the subject is, the larger the accumulated value ofan image block including the motion is. Block difference accumulator 24detects an accumulated value, which is calculated for each image block,as a motion amount of the subject. Block difference accumulator 24outputs a motion amount detected for each image block, to motionblending ratio calculator 12.

Motion blending ratio calculator 12 calculates a blending ratio based ona motion amount detected for each image block. The motion blending ratiois a ratio of blending each of the pixels in the long exposure image anda co-located one of the pixels in the corrected short exposure image.Motion blending ratio calculator 12 outputs the calculated motionblending ratio to motion-adapted image synthesizer 14.

Here, a procedure of calculating a motion blending ratio is discussedbelow. First, motion blending ratio calculator 12 smoothes motionamounts between image blocks. Specifically, motion blending ratiocalculator 12 fragments differences in motion amount between neighboringimage blocks and interpolates the motion amounts of pixels in such amanner that the differences in motion amount are allocated according todistances between respective pixels and the barycenter of the imageblock. Subsequently, motion blending ratio calculator 12 calculates amotion blending ratio for pixels co-located in the long exposure imageand the corrected short exposure image in such a manner that a blendingratio of blending the corrected short exposure image with the longexposure image has a positive correlation with the motion amount of thesubject. This is because a subject image in the long exposure image ishighly likely to blur in an image block where a large motion amount ofthe subject is detected.

As a result of the smoothing, a change in motion amount between pixelsis smoothed across the boundary of the image blocks, and thus thecalculated motion blending ratio between the pixels also smoothlychanges across the boundary between the image blocks. In other words,the motion blending ratio of the long exposure image and the correctedshort exposure image is determined for each pixel instead of each imageblock. This can reduce blockiness in a post-synthesis motion-adaptedimage (to be described later).

Motion-adapted image synthesizer 14 synthesizes the long exposure imageoutput from image sensor 4 and the corrected short exposure image outputfrom level adjuster 8, based on the motion blending ratio from motionblending ratio calculator 12, to generate a motion-adapted image.Specifically, motion-adapted image synthesizer 14 performs alphablending on each of the pixels of the long exposure image and theco-located one of the pixels of the corrected short exposure image byusing the motion blending ratio of the pixels as a coefficient. At thispoint, in an area of a subject image with large blur in the longexposure image, the corrected short exposure image is blended with thelong exposure image at a high motion blending ratio. This can correctmotion blur of the subject image in the long exposure image. In an areawhere a subject image in the long exposure image with small motion blur,the corrected short exposure image is blended with the long exposureimage at a low motion blending ratio. The blending is performed to avoidunnecessary deterioration in image quality in an area where a subjectimage has small motion blur in the long exposure image. This is becausethe S/N of the corrected short exposure image is lower than that of thelong exposure image. Motion-adapted image synthesizer 14 outputs thegenerated motion-adapted image to HDR image synthesizer 18.

Based on the long exposure image and the short exposure image outputfrom image sensor 4, luminance blending ratio calculator 16 calculates aluminance blending ratio of blending the pixels co-located in the shortexposure image and the long exposure image. Luminance blending ratiocalculator 16 outputs the calculated luminance blending ratio to HDRimage synthesizer 18.

HDR image synthesizer 18 synthesizes the motion-adapted image outputfrom motion-adapted image synthesizer 14 and the short exposure imageoutput from image sensor 4, based on the luminance blending ratio fromluminance blending ratio calculator 16, to generate an HDR image. HDRimage synthesizer 18 performs alpha blending on the pixel of themotion-adapted image and the co-located pixel of the short exposureimage by using the luminance blending ratio of the pixels as acoefficient. This can generate an HDR image adaptive for both a motionof the subject and the luminance of the pixel in the long exposure imageand the luminance of the pixel in the short exposure image.

1-2. Operations of Image Processor

An operation performed by image processor 2 according to Embodiment 1 isdescribed with reference to FIGS. 3 to 4E. FIG. 3 is a flow chartindicating a flow of the operation performed by image processor 2according to Embodiment 1. FIG. 4A is a schematic view of an example oflong exposure image 28 according to Embodiment 1. FIG. 4B is a schematicview of an example of short exposure image 30 according to Embodiment 1.FIG. 4C is a schematic view of an example of corrected short exposureimage 32 according to Embodiment 1. FIG. 4D is a schematic view of anexample of a motion amount distribution of subject 26 detected by motionamount detector 10 according to Embodiment 1. FIG. 4E is a schematicview of an example of HDR image 34 generated by image processor 2according to Embodiment 1.

As illustrated in FIG. 3, sensor controller 6 first determines a firstexposure time and a first sensor gain (first exposure sensitivity) and asecond exposure time and a second sensor gain (second exposuresensitivity) for image sensor 4 according to the brightness of subject26 (see FIGS. 4A and 4B) (S101). Image sensor 4 outputs short exposureimage 30 of subject 26 captured with the first exposure time and thefirst sensor gain determined by sensor controller 6, and outputs longexposure image 28 of subject 26 captured with the second exposure timeand the second sensor gain (S102).

For example, image sensor 4 captures images of subject 26 as illustratedin FIGS. 4A and 4B. In the examples of FIGS. 4A and 4B, the close viewof subject 26 shows a crossing flag held by a hand of a person crossinga road and the distance view of subject 26 shows a building facing aroad. At this time, the crossing flag held by the hand of the person isslightly raised.

Image sensor 4 outputs, for example, long exposure image 28 asillustrated in FIG. 4A. As illustrated in FIG. 4A, long exposure image28 is captured in the second exposure time longer than the firstexposure time, and thus long exposure image 28 is entirely brighter thanshort exposure image 30 (see FIG. 4B). In long exposure image 28, animage of moving subject 26 (the hand of the person and the crossingflag) blurs but an image of fixed subject 26 (e.g., the building) isclearly captured. In FIG. 4A, an edge of the blurred image of subject 26is indicated by broken lines, whereas an edge of the clear image ofsubject 26 is indicated by solid lines.

Moreover, image sensor 4 outputs, for example, short exposure image 30as illustrated in FIG. 4B. As illustrated in FIG. 4B, short exposureimage 30 is captured with the first exposure time shorter than thesecond exposure time, and thus short exposure image 30 is entirelydarker than long exposure image 28 (see FIG. 4A), and almost no motionblur occurs in the overall image of subject 26. Returning to FIG. 3,level adjuster 8 adjusts the luminance level of short exposure image 30output from image sensor 4 to substantially match the luminance level oflong exposure image 28, to generate corrected short exposure image 32from short exposure image 30 (S103).

Level adjuster 8 generates, for example, corrected short exposure image32 as illustrated in FIG. 4C. As illustrated in FIG. 4C, corrected shortexposure image 32 is approximately as bright as long exposure image 28and is entirely brighter than short exposure image 30 before theluminance level is adjusted. It is to be noted that noise may occur inoverall corrected short exposure image 32 when the luminance level israised. In FIG. 4C, dots indicate the noise in corrected short exposureimage 32.

Returning to FIG. 3, motion amount detector 10 detects a motion amountof subject 26 for each image block of long exposure image 28 (orcorrected short exposure image 32) by calculating a difference betweenlong exposure image 28 output from image sensor 4 and corrected shortexposure image 32 output from level adjuster 8 (S104).

Motion amount detector 10 detects a motion amount of subject 26 for eachimage block as illustrated in, for example, FIG. 4D. In (a) in FIG. 4D,each square indicates an image block. In the example of (a) in FIG. 4D,an image block having a large motion amount becomes brighter.Specifically, an image block becomes brighter as a difference between apixel value in long exposure image 28 and a pixel value in correctedshort exposure image 32 increases. In FIG. 4D, the motion amount isindicated by the density of shading. The larger the motion amount is,the lower the density of shading is.

Returning to FIG. 3, motion blending ratio calculator 12 calculates amotion blending ratio by smoothing the motion amounts of image blocks(S105). Thus, as illustrated in (b) in FIG. 4D, the motion amount moresmoothly changes across a boundary between the image blocks than in (a)in FIG. 4D before the smoothing.

Next, motion-adapted image synthesizer 14 generates a motion-adaptedimage by synthesizing long exposure image 28 output from image sensor 4and corrected short exposure image 32 output from level adjuster 8,based on the motion blending ratio from motion blending ratio calculator12 (S106). In this way, pixels in corrected short exposure image 32 areblended with pixels constituting a small-blur image of subject 26 inlong exposure image 28 at a low motion blending ratio, whereas pixels incorrected short exposure image 32 are blended with pixels constituting alarge-blur image of subject 26 in long exposure image 28 at a highmotion blending ratio. In an area where an image of subject 26 does notblur at all in long exposure image 28, the pixels of long exposure image28 having a higher S/N may be used as they are without blending of thepixels in corrected short exposure image 32.

Based on long exposure image 28 and short exposure image 30 output fromimage sensor 4, luminance blending ratio calculator 16 calculates aluminance blending ratio (S107). HDR image synthesizer 18 synthesizesshort exposure image 30 output from image sensor 4 and themotion-adapted image output from motion-adapted image synthesizer 14,based on the luminance blending ratio from luminance blending ratiocalculator 16, to generate HDR image 34 (S108).

HDR image synthesizer 18 generates, for example, HDR image 34 asillustrated in FIG. 4E. As illustrated in FIG. 4E, in HDR image 34, alarge-blur image of subject 26 (a hand of a person and a crossing flag)in long exposure image 28 is replaced with an image of subject 26 (thehand of the person and the crossing flag) in corrected short exposureimage 32. At this time, noise stemming from corrected short exposureimage 32 slightly occurs in the image of the hand of the person and thecrossing flag in HDR image 34. In FIG. 4E, dots indicate the noise inHDR image 34.

1-3. Example and Comparative Example

An image processor according to a comparative example is described withreference to FIGS. 5 to 7. FIG. 5 is a diagram for explaining processingperformed by a sensor controller of the image processor according to thecomparative example. FIG. 6 is a diagram illustrating a long exposureimage, a corrected short exposure image, and an HDR image which aregenerated by the image processor according to the comparative examplewhen the brightness of the subject is illuminance L4. FIG. 7 is adiagram illustrating a long exposure image, a corrected short exposureimage, and an HDR image which are generated by the image processoraccording to the comparative example when the brightness of the subjectis illuminance L5.

As illustrated in (a) and (b) in FIG. 5, when the brightness of thesubject decreases from illuminance L3, in the image processor accordingto the comparative example, the sensor controller controls a firstexposure sensitivity so that the first exposure time changes fromincreasing to constant (T4 (>T1)) and the first sensor gain changes fromconstant (G1) to increasing. In addition, as illustrated in (a) and (b)in FIG. 5, when the brightness of the subject decreases from illuminanceL3, the sensor controller controls a second exposure sensitivity so thatthe second exposure time changes from increasing to constant (T3) andthe second sensor gain changes from constant (G1) to increasing. Each ofpreset limit values for the first sensor gain and the second sensor gainis G3. It is to be noted that, as illustrated in (a) and (b) in FIG. 5,the graph of the first sensor gain and the graph of the second sensorgain match each other.

FIG. 6 shows the results of imaging of the subject performed by theimage processor according to the comparative example in the case wherethe image sensor is moved horizontally at 16°/second with respect to thesubject in a state in which the brightness of the subject is illuminanceL4 (approximately 50 Lux: low illuminance). As illustrated in (a) inFIG. 6, large motion blur occurred in a long exposure image (with secondexposure time: T3 and second sensor gain: G4) generated by the imageprocessor according to the comparative example. As illustrated in (b) inFIG. 6, medium motion blur occurred in a corrected short exposure image(with first exposure time: T4 and first sensor gain: G4) generated bythe image processor according to the comparative example. In this way,as illustrated in (c) in FIG. 6, medium motion blur stemming from thecorrected short exposure image occurred in an HDR image generated by theimage processor according to the comparative example.

FIG. 7 shows the results of imaging of the subject performed by theimage processor according to the comparative example in the case wherethe image sensor is moved horizontally at 16°/second with respect to thesubject in a state in which the brightness of the subject is illuminanceL5 (approximately 6 Lux: extremely low illuminance). As illustrated in(a) in FIG. 7, large motion blur and medium noise occurred in a longexposure image (with second exposure time: T3 and second sensor gain: G5(>G4)) generated by the image processor according to the comparativeexample. In addition, as illustrated in (b) in FIG. 7, medium motionblur and large noise occurred in a corrected short exposure image (withfirst exposure time: T4 and first sensor gain: G5) generated by theimage processor according to the comparative example. In this way, asillustrated in (c) in FIG. 7, medium motion blur stemming from thecorrected short exposure image and large noise occurred in an HDR imagegenerated by the image processor according to the comparative example.

Next, an image processor according to an example is described withreference to FIGS. 2, 8 and 9. FIG. 8 is a diagram illustrating a longexposure image, a corrected short exposure image, and an HDR image whichare generated by the image processor according to the example when thebrightness of a subject is illuminance L4. FIG. 9 is a diagramillustrating a long exposure image, a corrected short exposure image,and an HDR image which are generated by the image processor according tothe example when the brightness of the subject is illuminance L5.

In the image processor according to the example, sensor controllercontrols a first exposure sensitivity and a second exposure sensitivityas illustrated in (a) and (b) in FIG. 2.

FIG. 8 shows the results of imaging of the subject performed by theimage processor according to the example in the case where the imagesensor is moved horizontally at 16°/second with respect to the subjectin a state in which the brightness of the subject is illuminance L4 (lowilluminance). As illustrated in (a) in FIG. 8, large motion bluroccurred in the long exposure image (with second exposure time: T3 andsecond sensor gain: G4) generated by the image processor according tothe example. As illustrated in (b) in FIG. 8, almost no motion bluroccurred in the corrected short exposure image (with first exposuretime: T1 (<T4) and first sensor gain: G6 (>G4)) generated by the imageprocessor according to the example. In this way, as illustrated in (c)in FIG. 8, almost no motion blur stemming from the corrected shortexposure image occurred in the HDR image generated by the imageprocessor according to the example.

FIG. 9 shows the results of imaging of the subject performed by theimage processor according to the example in the case where the imagesensor is moved horizontally at 16°/second with respect to the subjectin a state in which the brightness of the subject is illuminance L5(extremely low illuminance). As illustrated in (a) in FIG. 9, largemotion blur and medium noise occurred in a long exposure image (withsecond exposure time: T3 and second sensor gain: G5) generated by theimage processor according to the example. As illustrated in (b) in FIG.9, comparatively large motion blur occurred but only medium noiseoccurred in a corrected short exposure image (with first exposure time:T5 (>T4) and first sensor gain: G2 (<G5)) generated by the imageprocessor according to the example. In this way, as illustrated in (c)in FIG. 9, comparatively large motion blur stemming from the correctedshort exposure image occurred but only medium noise occurred in an HDRimage generated by the image processor according to the example.

In view of the above results, it is confirmed that the visibility of theimage is increased because reduced motion blur occurred in the HDR image((c) in FIG. 8) in the example when the moving subject is imaged underlow illumination (illuminance L4) compared to the motion blur occurredin the HDR image according to the comparative example (c) in FIG. 6). Inaddition, it is confirmed that the visibility of the image is increasedbecause less noise occurred in the HDR image ((c) in FIG. 9) in theexample when the moving subject is imaged under extremely lowillumination (illuminance L5) compared to the noise occurred in the HDRimage according to the comparative example (c) in FIG. 7).

1-4. Advantageous Effects

Image processor 2 according to this embodiment is capable of reducingmotion blur in the HDR image because the first exposure time is kept tobe short, for example, even when a moving subject is imaged under lowilluminance. In addition, the first sensor gain is kept to be small, forexample, even when the moving subject is imaged under extremely lowilluminance. Thus, it is possible to reduce noise in the HDR image.

Embodiment 2 2-1. Configuration of Image Processor

A configuration of image processor 2A according to Embodiment 2 isdescribed with reference to FIGS. 10 and 11. FIG. 10 is a block diagramillustrating a configuration of image processor 2A according toEmbodiment 2. FIG. 11 is a diagram for illustrating processing performedby sensor controller 6A of image processor 2A according to Embodiment 2.It is to be noted that, in this embodiment, the same constituentelements as those in Embodiment 1 are assigned with the same signs, andare not described repeatedly.

As illustrated in FIG. 10, image processor 2A according to Embodiment 2is different from image processor 2 in Embodiment 1 in the control of afirst exposure sensitivity performed by sensor controller 6A. It is tobe noted that the control of a second exposure sensitivity performed bysensor controller 6A is the same as in Embodiment 1.

As illustrated in (a) and (b) in FIG. 11, when the brightness of asubject decreases from illuminance L1, sensor controller 6A controls thefirst exposure sensitivity so that the first exposure time changes fromincreasing to constant (T1) and the first sensor gain changes fromconstant (G1) to increasing. In addition, when the brightness of thesubject decreases from illuminance L2, sensor controller 6A controls thefirst exposure sensitivity so that the first exposure time changes fromconstant (T1) to increasing and the first sensor gain changes fromincreasing to constant (G2).

Furthermore, when the brightness of a subject decreases from illuminanceL5 (one example of a fourth brightness) lower than illuminance L2,sensor controller 6A controls the first exposure sensitivity so that thefirst exposure time changes from increasing to constant (T5) and thefirst sensor gain changes from constant (G2) to increasing. In thiscase, a preset limit value for the first exposure time is T5, and apreset limit value for the first sensor gain is G5. It is to be notedthat, in the examples illustrated in (a) and (b) in FIG. 11, illuminanceL5 is approximately 6 Lux.

2-2. Advantageous Effects

As described in Embodiment 1, when the brightness of the subjectdecreases from illuminance L2, the first exposure time changes fromconstant to increasing and the first sensor gain changes from increasingto constant. For this reason, for example, when the moving subject isimaged under extremely low illuminance, noise in the HDR image isreduced, but motion blur may occur in the HDR image because the firstexposure time becomes longer.

In comparison, when the brightness of the subject decreases fromilluminance L5, image processor 6A according to this embodiment performscontrol so that the first exposure time changes from increasing toconstant. In this way, increase in the first exposure time is reduced.Thus, it is possible to reduce motion blur in the HDR image.

Other Variations

The image processors according to one or more aspects have beendescribed above based on the embodiments described above. However, thepresent disclosure is not limited to the embodiments. The one or moreaspects may encompass embodiments obtainable by making variousmodifications that a person skilled in the art may arrive at to any ofthe embodiments and embodiments obtainable by combining any of theconstituent elements in the different embodiments within the scope ofthe present disclosure.

In Embodiment 2, when the brightness of the subject decreases fromilluminance L3, sensor controller 6A controls the second exposuresensitivity so that the second exposure time changes from increasing toconstant (T3) and the second sensor gain changes from constant (G1) toincreasing. However, such control is not limited thereto. For example,when the brightness of the subject decreases from illuminance L6 (notillustrated) higher than illuminance L1, sensor controller 6A maycontrol the second exposure sensitivity so that the second exposure timechanges from increasing to constant (T2) and the second sensor gainchanges from constant (G1) to increasing.

Furthermore, when the brightness of the subject decreases fromilluminance L5, sensor controller 6A may control the second exposuresensitivity so that the second exposure time changes from constant (T2)to increasing and the second sensor gain changes from increasing toconstant (G7: not illustrated).

In this case, a preset limit value for the second exposure time is T6(not illustrated).

It is to be noted that each of the constituent elements in each of theembodiments may be implemented by being configured using dedicatedhardware or by means of a software program suitable for the constituentelement being executed. Each of the constituent elements may beimplemented by means of a program executer such as a CPU or a processorreading and executing a software program recorded on a recording mediumsuch as hard disc or semiconductor memory.

Alternatively, a part or all of the functions of the image processoraccording to any of the embodiments may be implemented by means of aprocessor such as a CPU executing a program.

A part or all of the constituent elements of any of the image processorsmay be configured as an IC card which can be attached to and detachedfrom the image processor or as a stand-alone module. The IC card or themodule is a computer system configured with a microprocessor, a ROM, aRAM, and so on. The IC card or the module may also be the aforementionedsuper-multi-function LSI. The IC card or the module achieves itsfunctions through the microprocessor's operations according to thecomputer program. The IC card or the module may have atamper-resistance.

The present disclosure may be implemented as the methods describedabove. The present disclosure may be implemented as computer programsfor executing these methods, using a computer, and may also beimplemented as digital signals including the computer programs.Furthermore, the present disclosure may also be implemented as computerprograms or digital signals recorded on computer-readable recordingmedia such as a flexible disc, a hard disk, a CD-ROM, an MO, a DVD, aDVD-ROM, a DVD-RAM, a BD (Blu-ray Disc), and a semiconductor memory. Inaddition, the present disclosure may also be implemented as the digitalsignals recorded on these recording media. Furthermore, the presentdisclosure may also be implemented as the computer programs or thedigital signals transmitted via a telecommunication line, a wireless orwired communication line, a network represented by the Internet, a databroadcast, and so on. In addition, the present disclosure may also beimplemented as a computer system including a microprocessor and memory,in which the memory stores the computer programs and the microprocessoroperates according to the computer programs. Furthermore, the programsor the digital signals may be performed by another independent computersystem by transmitting the programs or the digital signals recorded onthe recording media, or by transmitting the programs or digital signalsvia the aforementioned network and the like.

INDUSTRIAL APPLICABILITY

The present disclosure is applicable to image processors, etc. forgenerating an HDR image by, for example, synthesizing a plurality ofimages captured in different exposure times.

The invention claimed is:
 1. An image processor which generates a highdynamic range (HDR) image of a subject, the image processor comprising:an image sensor which outputs, in one-frame time, (i) a first image ofthe subject captured with a first exposure time and a first sensor gainand (ii) a second image of the subject captured with a second exposuretime and a second sensor gain, the second exposure time being longerthan the first exposure time; a sensor controller which, when abrightness of the subject changes, (i) controls a first exposuresensitivity so that the first image has a first image brightness and(ii) controls a second exposure sensitivity so that the second image hasa second image brightness, the first exposure sensitivity being aproduct of the first exposure time and the first sensor gain, the secondexposure sensitivity being a product of the second exposure time and thesecond sensor gain; a level adjuster which adjusts a luminance level ofthe first image to substantially match a luminance level of the secondimage, to generate a corrected image from the first image; a motionamount detector which detects a motion amount of the subject based on adifference in pixel value between pixels co-located in the correctedimage and the second image; a motion blending ratio calculator whichcalculates a motion blending ratio based on the motion amount, themotion blending ratio being a ratio between the corrected image and thesecond image when the corrected image and the second image are blended;a motion-adapted image synthesizer which synthesizes the corrected imageand the second image based on the motion blending ratio, to generate amotion-adapted image; and an HDR image synthesizer which synthesizes themotion-adapted image and the first image, to generate the HDR image,wherein the sensor controller: controls the first exposure sensitivityso that the first exposure time changes from increasing to constant andthe first sensor gain changes from constant to increasing, when thebrightness of the subject decreases from a first subject brightness; andcontrols the first exposure sensitivity so that the first exposure timechanges from constant to increasing and the first sensor gain changesfrom increasing to constant, when the brightness of the subjectdecreases from a second subject brightness lower than the first subjectbrightness.
 2. The image processor according to claim 1, wherein thesensor controller controls the second exposure sensitivity so that thesecond exposure time changes from increasing to constant and the secondsensor gain changes from constant to increasing, when the brightness ofthe subject decreases from a third subject brightness lower than thefirst subject brightness.
 3. The image processor according to claim 1,wherein the sensor controller controls the first exposure sensitivity sothat the first exposure time changes from increasing to constant and thefirst sensor gain changes from constant to increasing, when thebrightness of the subject decreases from a fourth subject brightnesslower than the second subject brightness.
 4. An image processing methodfor generating a high dynamic range (HDR) image of a subject, the imageprocessing method comprising: (a) outputting, in one-frame time, (i) afirst image of the subject captured with a first exposure time and afirst sensor gain and (ii) a second image of the subject captured with asecond exposure time and a second sensor gain, the second exposure timebeing longer than the first exposure time; (b) when a brightness of thesubject changes, (i) controlling a first exposure sensitivity so thatthe first image has a first image brightness and (ii) controlling asecond exposure sensitivity so that the second image has a second imagebrightness, the first exposure sensitivity being a product of the firstexposure time and the first sensor gain, the second exposure sensitivitybeing a product of the second exposure time and the second sensor gain;(c) adjusting a luminance level of the first image to substantiallymatch a luminance level of the second image, to generate a correctedimage from the first image; (d) detecting a motion amount of the subjectbased on a difference in pixel value between pixels co-located in thecorrected image and the second image; (e) calculating a motion blendingratio based on the motion amount, the motion blending ratio being aratio between the corrected image and the second image when thecorrected image and the second image are blended; (f) synthesizing thecorrected image and the second image based on the motion blending ratio,to generate a motion-adapted image; and (g) synthesizing themotion-adapted image and the first image, to generate the HDR image,wherein in (b): the first exposure sensitivity is controlled so that thefirst exposure time changes from increasing to constant and the firstsensor gain changes from constant to increasing, when the brightness ofthe subject decreases from a first subject brightness; and the firstexposure sensitivity is controlled so that the first exposure timechanges from constant to increasing and the first sensor gain changesfrom increasing to constant, when the brightness of the subjectdecreases from a second subject brightness lower than the first subjectbrightness.